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(57) Abstract 

The invention relates to a remote control apparatus for controlling a display apparatus. The remote control comprises an image sensor 
(7) and imaging optics (8) for projecting an image, for example of the display screen (2), on the image sensor. If a user moves the remote 
control apparatus, the projected image will move accordingly. From the movement of the projected image, a relative movement of the 
remote control apparatus with respect to the display apparatus is deduced, and converted to control commands which are transmitted to the 
display apparatus. 
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Remote control for display apparatus. 



FIELD OF THE INVENTION 

The invention relates to a remote control apparatus for controlling a display 
apparatus, comprising orientation means for detecting a relative movement of the remote 
control apparatus with respect to the display apparatus. 
5 The invention further relates to a system of a display apparatus and the above 

remote control apparatus. 

The invention fiirther relates to a display apparatus for use in the above system. 

BACKGROUND OF THE INVENTION 

1 0 With the advent of advanced features for television, such as Internet browsing, 

home shopping and game playing, the user must be enabled to control these new features from 
the usual viewing distance. For that reason, a remote control is needed which integrates the 
fimctions of a conventional remote control and a pointing device, such as a computer mouse. 
Various solutions have been proposed for this integration. Some of them are based on the 

1 5 technology applied in a light-pen, wherein the moment of detecting a light spot caused by the 
cathode ray of the display apparatus is compared with the synchronization signals. This 
solution is only applicable within a limited distance from the display screen. Other solutions 
are based on an infrared signal which is transmitted by the display apparatus and received by 
the remote control. For example, US 5,644,126 discloses a system of a display apparatus and a 

20 . remote control, the display apparatus having a light source, e.g. a photodiode, the light of 
which is received by the remote control and led through a lattice. The resultant interference 
pattern is projected onto an array sensor. If the remote control is moved, the interference 
pattern shifts, which can be detected by means of the array sensor. The movement is then 
converted to appropriate commands for controlling fimctions of the display apparatus, e.g. 

25 choosing a menu option from an on-screen display menu. 

A disadvantage of the known systems is that both the remote control and the 
display apparatus need to be adapted to achieve the desired fimctionality. Hence, they do not 
provide a remote control which can be applied as an add-on to conventional display 
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apparatuses. Furthennore, only relative movements can be detected with the system of US 
5,644,126. 



OBJECT AND SUMMARY OF THE INVENTION 
5 It is an object of the invention to provide a remote control apparatus of the type 

defined in the opening paragraph which evades the disadvantages of the remote control 
apparatuses of the prior art. To this end, the remote control apparatus according to the 
invention is characterized in that the orientation means comprise an unage sensor and imaging 
optics for projecting an image on the image sensor, the orientation means being adapted to 

1 0 deduce said movement from a movement of the projected unage. The projected image may be 
of any object, e.g. the display screen, a window or a lamp. If the user moves the remote control 
apparatus, the projected image will move accordingly. The movement of the projected image 
can be detected, e.g. by comparing subsequent frames obtained from the image sensor. 
Software algorithms for detecting and analyzing the movement of the projected image are 

1 5 known per se and will only be discussed in so far as they are relevant for disclosing the 
invention. Background information may be obtained from, for example: 

A method for processing laser speckle images to extract high-resolution motion, 
Andrew Houghton, Graham Rees and Peter Ivey Measurement Science & Technology no 8 
(1997), pages 61 1-617 lOP publishmg Ltd. 

20 - Linear and rotary encoders using electronic speckle correlation, Ichirou 

Yamagouchi and Tadashige Fujita, Optical Engineering / December 1991 / vol. 30 No. 12, 
pages 1862- 1868. 

The movements of the projected image are then translated into appropriate 
commands for controlling ftmctions of the display apparatus, e.g. a cursor position. An 

25 additional advantage of the invention is that rotation can be detected as well. 

An embodiment of the remote control apparatus according to the invention is 
characterized in that the remote control apparatus further comprises activation means for 
activating the orientation means m response to a user command or predetermined event. The 
activation means may comprise a grip or tilt sensor for detecting the event of picking up the 

30 remote control apparatus. Alternatively, the activation means may comprise a button which 
has to be pressed intentionally to activate the orientation means. Upon activation, the remote 
control apparatus may perform initialization actions, e.g. to provide a default cursor position. 
AH subsequent movements will then be translated into corresponding cursor movements. 
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A further embodiment is characterized in that the remote control apparatus 
further comprises distance measuring means for measuring a distance to an object of which an 
image is projected on the image sensor, the orientation means being adapted to adjust said 
relative movement with respect to said distance. For example, the remote control apparatus 

5 may be equipped with an auto-focus unit. On the one hand, the auto-focus xmit guarantees a 
sharp projected picture on the image sensor independent of the distance to the projected object. 
On the other hand, the auto-focus unit may provide information with respect to the distance 
between the remote control apparatus and said object, which information may be taken mto 
account when determining the sensitivity of the orientation means. For example, when aiming 

10 the remote control apparatus at a nearby object, a movement of the remote control apparatus 
will cause a relatively large movement of the projected image. In that case, the movements 
may be translated into relatively small cursor movements in comparison to the situation of 
aiming at a remote object. Techniques for auto-focussing are well known, see for example: 
Digital method for measuring the focus error, by Adolf W. Lohmann, David 

1 5 Mendiovic, and Zeev Zalevsky APPLIED OPTICS / vol. 36. No. 28 / 1 October 1997, pages 
7204-7209, 

Speckle techniques for absolute distance measurement, G.E. Hege, H.J. Tiziani, 
Tech. Messen 54 (6) pages 237 - 242 (1987). 

A further embodiment is characterized in that the orientation means are further 
20 adapted to recognize an image of the display apparatus. As mentioned before, the display 

apparatus itself may be conveniently used as an object to aim at. Generally, the display screen 
is salient in the viewing environment because of its light production. For the purpose of 
controlling objects displayed on the display screen, specific properties of the display screen 
may advantageously be taken into account. For example, the dimensions of the screen and/or 
25 the picture refresh rate may be used to distinguish the screen from the environment. 

A furdier embodiment is characterized in that the orientation means are further 
adapted to store properties (dimensions, refresh rate) of the display apparatus. By explicitly 
storing such properties, e.g. by downloading or explicitly entering them by means of the 
buttons, the orientation means have a priori knowledge about the projected image of the 
30 display apparatus. Hence, confusion caused by, for example, the flickering of lamps or the 
dimensions of windows can be avoided. 

A further embodiment is characterized in that the orientation means are adapted 
to recognize an icon with predetermmed properties on the display of the display apparatus. In 
this way it is achieved that the display screen can be recognized unambiguously, because the 
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icon can be made unique by means of a specific shape or color combination. The icon may be 
located in a fixed position, or it may double as the screen cursor. 

A fiirther embodiment is characterized in that said predetermined properties 
comprise a periodic change of predetermined parameters (e.g. color, brightness). In this way, 
5 the unambiguous recognition of the display apparatus is fiirther simplified. 

A fiirther embodiment is characterized in that the remote control is adapted to 
decode status information related to the display apparatus from said periodic change. The 
periodic change of the previous embodiment may be modulated with signals representing said 
status information. The status information may comprise sensitivity settings, properties of the 
10 display apparatus etc. 

An embodiment is characterized in that the orientation means are fiirther 
adapted to determine a relative position of the remote control apparatus with respect to the 
display apparatus by deducing said relative position from the projected image of the display 
apparatus, hi this way, not only relative positioning, but also absolute positioning becomes 
1 5 feasible, because after recognition of the display screen the necessary properties, such as 
screen dimensions, are available to the orientation means. 

The invention is particularly suitable for television receivers, monitors, 
(game)computers and presentation devices. 

20 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other aspects of the invention are apparent firom and will be 
elucidated, by way of a non-limitative example, with reference to the embodiment(s) described 
hereinafter. In the drawings, 
25 Figure 1 schematically shows a remote control apparatus and a display 

apparatus according to the invention. 

Figure 2 shows a block diagram of the relevant elements of a remote control 
apparatus according to the invention, 

Figure 3 shows various types of movement which are detected by a remote 
30 control apparatus according to the invention, 

Figure 4 schematically shows absolute cursor positioning by means of a remote 
control apparatus according to the invention, 

Figure 5 shows an example of information transfer from a display apparatus to 
a remote control apparatus according to the invention. 
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DESCRffTION OF EMBODIMENTS 

Figure 1 shows a television receiver 1 comprising a display screen 2, and a 
remote control 3. The remote control 3 comprises a circuit board 4 and buttons 5 which have 
5 the same function as the buttons of a computer mouse. Additionally, conventional remote 
control buttons may be provided, e.g. numerical keys, teletext keys etc. The remote control 3 
further comprises a grip sensor 6 for detecting when the remote control is picked up by a user. 
Imaging optics 8 are capable of projecting an image of a remote object, e.g. the television 
receiver 1, onto an image sensor 7 which is placed on tiie circuit board 4. A LED (Light 

1 0 Emitting Diode) 9 is mounted on the circuit board 4 for transmitting infrared light signals to 
the television receiver 1 . An auto-focus unit 1 0 is provided for establishing a sharp projection 
upon the image sensor 7 and for measuring the distance to the projected object. 

Figure 2 shows a block diagram of the relevant elements of the remote control 
3. An image of the display screen 2 is projected onto the image sensor 7, which is a matrix of 

1 5 photo-sensitive cells, e.g. photo-diodes or integrating photo-cells. The photo-sensitive cells 
(pixels) are scanned periodically by addressing them by means of a pixel addressing mit 13. 
The resultant electric signal is converted by an A/D converter 14 and transmitted to a signal 
processor 12 in digital form. The signal processor 12, constituting orientation means for 
detecting a relative movement with respect to a display apparatus to be controlled, is capable 

20 of extracting salient distinctive elements from the available information. Software algorithms 
for detecting such elements are known per se. For example, the signal processor 12 may look 
for transitions between light and dark areas whose dimensions and contrast values exceed a 
certain threshold. Suitable elements the signal processor 12 may look for are the comers of the 
display screen 2, a cursor or other elements of the displayed content. Alternatively, correlation 

25 techniques may be used for the extraction of salient elements. These techniques are known per 
se. 

The coordinates of said elements are stored in a memory (not shown). By 
storing the coordinates of at least two elements, both translation along two coordinate axes and 
rotation can be detected. Periodically, e.g. every frame or averaged over a number of frames, 
30 the measured translation or rotation will be converted to appropriate commands for controlling 
the television receiver 1, and transmitted to the television receiver 1 by means of infrared 
signals generated by the LED 9, or by means of an altemative transmission method. The signal 
processor 12, or another dedicated circuit, generates the codes to be transmitted fi-om the 
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computed translation and rotation, together with the state of the buttons 5 and the grip sensor 
6. 

The grip sensor activates signal processor 12 when the remote control 3 is 
operated by the user. Upon activation an initialization routine is started which performs the 
5 following functions: 

Transmitting an initialization code to the television receiver 1 , causing a cursor 
and/or menu to be activated. 

Capturing an image from the image sensor 

Determination of the coordinates of salient elements of the image, and 
1 0 computing relative translation or rotation by means of correlation techniques. 

For the recognition of image elements it is not necessary to take all pixels into account. Only 
the pixels close to such an element need to be stored and processed, thus saving processing 
power. Dependent on the required response times, detected movements may be averaged over 
multiple measurements before transmitting corresponding commands to the television receiver 
15 1. 

The recognition of the display screen 2 may be enhanced by taking specific 
properties of the display screen 2 into accoimt. Generally, displays are characterized by a 
specific refresh rate, which ranges from 25 Hz (classical TV) to 160 Hz. This refresh rate can 
be detected, and this can be used to determine which information on the image sensor 7 is 
20 obtained from the display screen 2. From this information the salient elements can then be 
extracted. Elements consist of light-dark transitions with a certain contrast and dimensions. 
For a reliable recognition at least a few pixels should be involved. 

If the position of two elements at a time t is given by (xO,yO)(t) and (xl, yl)(t) 
respectively, translation and rotation with respect to point (xO, yO) can be calculated by 
25 determining the coordinates of the two points at two subsequent points in time tO and tl as, 
respectively, (xO, yO) (tO), (xl, yl) (tO) and (xO, yO) (tl), (xl. yl) (tl). 
Now translation is defined as: 

dx = xO(tl)-xO(tO) 

dy = yO(tl)-yO(tO) 
30 Rotation is defined as: 

© = arctan ((xl(tl) - xl(tO) - dx)/(yl(tl) - yl(tO) - dy)). 
Figure 3 shows the various degrees of freedom of the remote control 3. 
AU measures discussed can be applied without adapting the software or hardware of the 
display apparatus to be controlled. Additional reliability and ftmctionality may be obtained by 
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adapting the displayed content on the display screen 2 in a predetermined way. For example, a 
cursor may be hard to detect within the unpredictable content of the displayed picture. This 
problem may be solved by periodically changing some properties of the cursor, e.g. its 
intensity. By switching the cursor on and off at regular times, e.g. every other refresh cycle, 
5 the detector is able to clearly distinguish the cursor from the displayed picture. By filtering 
over time, the cursor position can be determined reliably. 

The absolute position with respect to the display screen 2 may be determined by 
comparing its position with the fixed positions of fiirther elements. Figure 4 shows a cursor 15 
having position (Xc, Yc) and two elements 16 and 18 at positions (X2,Y2) and P^1,Y1) 

1 0 respectively. The presence of a third element 1 7 at position PCO, YO) will enhance the 

reliability of the computations. By choosing fixed positions for the further elements, e.g. the 
comers of the display screen, the signal processor 12 is able to infer the screen diameter. The 
absolute position (X,Y) of the cursor 15 is defined as: 
X = xc/(x2-xl) 

15 - Y = yc/(y2-yl) 

The periodical change of cursor properties may be modulated in order to 
transmit information fi-om the display apparatus 1 to the remote control 3. For example, 
information related to the status of the display apparatus 1, or a desired change of sensitivity of 
tiie remote contiol 3, may be transferred to the remote control 3. Cursor modulation is 

20 achieved by contiroUing the periodic change with codes representing the information to be 
transmitted. 

This is illustrated in Figure 4. The step fiinction 19 represents the frame cycle, 
the high level representing the even frames, and the low level representing the odd frames. The 
step fimction 20 represents the cursor signal, the high level representmg 'cursor on', and the 
25 low level representing 'cursor off. Techniques for encoding information are well known per 
se. 



In siramiary, the invention relates to a remote control apparatus for controlling a 
display apparatus. The remote control comprises an image sensor and imaging optics for 
30 projecting an image, for example of the display screen, on the image sensor. If a user moves 
the remote control apparatus, the projected image will move accordingly. From the movement 
of the projected image, a relative movement of the remote control apparatus with respect to the 
display apparatus is deduced, and converted to control commands which are transmitted to the 
display apparatus. 
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Although the invention has been described with reference to particular 
illustrative embodiments, variants and modifications are possible within the scope of the 
inventive concept. 

The word 'comprising' does not exclude the presence of elements or steps other 
5 than those listed in a claim. 
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1 . A remote control apparatus for controlling a display apparatus, comprising 
orientation means for detecting a relative movement of the remote control apparatus with 
respect to the display apparatus, characterized in that the orientation means comprise an image 
sensor and imaging optics for projecting an image on the image sensor, the orientation ineans 

5 being adapted to deduce said movement from a movement of the projected image. 

2. A remote control apparatus as claimed in claim 1 , characterized in that the 
remote control apparatus fiirther comprises activation means for activating the orientation 
means in response to a user command or predetermined event. 

10 

3. A remote control apparatus as claimed in claim 1 or 2, characterized in that the 
remote control apparatus further comprises distance measuring means for measuring a distance 
to an object of which an image is projected on the image sensor, the orientation means being 
adapted to adjust said relative movement with respect to said distance. 

15 

4. A remote control apparatus as claimed in claim 1 or 3, characterized in that the 
orientation means are further adapted to recognize an image of the display apparatus. 

5. A remote control apparatus as claimed in claim 4, characterized in that the 
20 orientation means are fiirther adapted to store properties (dimensions, refresh rate) of the 

display apparatus. 

6. A remote control apparatus as claimed in claim 4 or 5, characterized in that the 
orientation means are adapted to recognize an icon with predetermined properties on the 

25 display of the display apparatus. 

7. A remote control apparatus as claimed in claim 6, characterized in that said 
predetermined properties comprise a periodic change of predetermined parameters (e.g. color, 
brightness). 
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8. A remote control apparatus as claimed in claim 7, characterized in that the 

remote control is adapted to decode status information related to the display apparatus from 

said periodic change. 

5 9. A remote control apparatus as claimed in anyone of claims 4 to 8, characterized 

in that the orientation means are ftirther adapted to determine a relative position of the remote 
control apparatus with respect to the display apparatus by deducing said relative position from 
the projected image of the display apparatus. 

10 10. A system of a display apparatus and a remote control apparatus having the 

features of the remote control apparatus as defined in claims 4 to 9, characterized in that the 
display apparatus is adapted to display an icon with predetermined properties for enhancing 
the recognizability of an image of the display apparatus when projected onto the image sensor 
of the remote control apparatus. 

15 

11. A display apparatus having the features of the display apparatus of the system 

as defined in claim 10. 
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